Archiving, Indexing and Accessing Web Materials: Solutions for large amounts of data

The archiving of Internet materials presents two major challenges: the dynamic nature of the content and the massive number of individual web pages. These two characteristics impact the choice of methods for storing, indexing and providing fast access to archives of materials retrieved from periodic web crawling activities. At the San Diego Supercomputer Center, we have applied two different technologies for archiving, indexing and accessing web archived materials for two projects: a Library of Congress / San Diego Supercomputer Center joint storage project (LOC/SDSC) and the National Science Digital Library (NSDL) persistent archive. In the LOC/SDSC project, we deployed the Wayback software for indexing and accessing a large archive collection from the Library of Congress. Our experience demonstrates that the Wayback software (release 0.6.0) is mature and suitable for handling small collections. Our modifications to the Wayback machine enabled the management of distributed collections. In the NSDL project, we used the Storage Resource Broker (SRB) data grid to index and archive web materials. This approach enabled the use of distributed and heterogeneous storage systems, while supporting access from multiple types of clients, ranging from web browsers, to workflow systems, to Perl and Python load libraries.